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CRQSS-REFERENC E TO REl.ATED APPIJCATIONS 

> This application is related to li.S. Applk^ition Serial No. entitled Mli THOD AND 

APPARATUS FOR DECODING ANGULAR ORIENTATION OF LATTICE CODES (Attorney 
Docket No. 07447. 0042-00000), and is related to U.S. Patent Application Serial No, entitled 
METHOD AND APPARATUS FOR DISPLAY OF SPATIALLY REGISTERED 
INFORMATION USING EMBEDDED DATA (Attorney Docket No. 07447.0044-00000), filed 
concurrently herewith. 

BAG KGROUND OF THE IN\ ENTION 

Apparatus, methods, and articles of manufacture consistent with the invention relate 
generally to spatial registration of infomiation, and in particular to capturing an image of first 
infonnation having embedded data, decoding the embedded data in the captured image, retrieving 
second information based on the decoding, and spatially registering the second information w^ith 
respect to the first information. 

It is often useful to register a first set of information with a second set of information. 
Augmented reality, for example, is implemented by creating a composite view of physical reality 
registered with computer-generated infomiation related to the physical reality. In an augmented 
reality system, a first set of infonnation, such as an image of physical reality, may be registered 
wMth a second set of information, such as information related to the image of physical reality. 

One conventional approach to creating augmented reality is to capture an image of a 
physical scene, identify objects in the scene using a scene recognition algorithm, retrieve 
infonnation based on the identified objects, and create a combined display of an image of the 
physical scene and infonnation related to the identified objects, thus augmenting the physical 
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scene. One drawback of such a sysleni is thai a large amount of processing power is required to 
execute the scene recognition algorithm, especially when it must differentiate between many 
different objects of a scene, and identify the location of the objects in the scene. Identifying the 
location of the objects allows the retrieved infonnation to be placed at a location in the composite 
5 display that is registered with the identified object. The retrieved mfonnation is "registered" by 
spatially relating the retrieved information to the identified object. Another drawback is that 
registration generally cannot be determined from a partial capture of the scene. 

What is needed is a system capable of registering first and second infomiation that does 
not suffer from the limitations of conventional systems. 

10 SUMMARY OF THE INVENTION 

Apparatus, methods, and articles of manufacture consistent w ith the present invention 
provide a registration scheme wherein a first set of infonnation on a substrate having embedded 
data embodied thereon is registered with a second set of information, based on the embedded data. 
In accordance with this registration scheme, an image capture device captures an image that 

15 includes embedded code and causes a display to be created that includes a combination of the 
captured image and infomiation that augments the image w ith additional infonnation. The 
additional information is registered with the captured image in the combined display. 

An apparatus consistent w ith the principles of the invention comprises an image capture 
device for capturing coded embedded data from a substrate having first image infonnation, a 

20 decoder for decoding the coded embedded data to develop registration infonnation, and a display 
for displaying second information registered w ith the first infonnation based on the registration 
infonnation. 
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A method consistent with the principles of the invention comprises capturing coded 
embedded data from a substrate ha\ ing llrst image information, decoding the coded embedded 
data to develop registration information, and displaying second infomiation registered with the 
first information based on the registration information. 
5 Additional objects and advantages of the invention will be set forth m part in the 

description which follows, and in part will be clear from the description, or may be learned by 
practice of the invention. The objects and advantages of the invention will be realized and 
attained by means of the elements and combinations particularly pointed out in the appended 
claims. It is to be understood that both the foregoing general description and the following 
10 detailed description are exemplary and explanatory only and are not restrictive of the invention, as 
claimed. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The accompanying draw ings, which are incorporated in and constitute a part of this 
specification, illustrate an embodiment of the invention and, together with the description, serve 
1 5 to explain the principles of the invention. 

Fig. 1 illustrates an overview of the properties of glyph marks and codes embodied in the 
glyph marks; 

Fig. 2 illustrates an embodiment of an image combining graphics and glyphs consistent 
with the present invention; 
20 Fig. 3 illustrates an enlarged view of a portion of the image illustrated in Fig. 2; 

Fig. 4 illustrates an image of a pictorial comprising glyphtones consistent with the 
principles of the present in\'ention; 
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Fig. 5 illustrates an example of a portion of a glyph address caqiet code with overlay tiled 
label code; 

Fig. 6 illustrates a system for reading an image having embedded data, decoding the 
embedded data in the image, and developing human-sensible infomiation based on the decoded 
5 embedded data; 

Fig. 7 illustrates a logical configuration of elements consistent with principles of the 
present invention; 

Fig. 8 illustrates another embodiment of a system consistent with the principles of the 
invention; 

10 Fig. 9 is another embodiment of a system built in accordance w ith the principles of the 

present invention; 

Fig. 10 is a diagram illustrating registration of infomiation consistent wqth the principles 
of the invention; 

Fig. 1 1 is a block diagram illustrating one embodiment of a lens apparatus consistent with 
1 5 the principles of the invention; 

Fig. 12 is a cutaw ay side view^ of the lens apparatus shown in Fig. 1 1 ; 
Fig. 1 3 illustrates an example of a substrate, an overlay image, and the substrate overlaid 
with the overlay image as seen through the lens viewport illustrated in Fig. 1 1 and Fig. 12; 

Fig. 14 illustrates another example of a substrate, an overlay image, and the substrate 
20 overlaid with the overlay image as seen through the lens viewport illustrated in Fig. 1 1 and Fig. 
12; 
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[Mg. 1 5 is a ilowchart showing the processing performed by compuler 4()() lo register an 

image; 

iMg. 16 graphically illustrates the process of creating an image of superimposed 
neighborhood images in a glyph lattice image, in accordance with the flowchart of Fig. 17; 
5 Fig. 1 7 is a flowchart showing the processing performed by a computer to determine a 

quadrant offset angle using a lattice image; 

Fig. 1 8 is a flowchart showing the processing performed by a computer to create a 
composite lattice image for use in detemiining a quadrant offset angle using a composite lattice 
image; 

10 Fig. 19 illustrates finding a glyph ccntroid; 

Fig. 20 and Fig. 21 illustrate analyzing the composite image, as illustrated in Fig. 16, to 
accurately detemiine the rotation angle of the composite image; 

Fig. 22 illustrates an embodiment of address codes encoded in a portion of a glyph address 

carpet; 

1 5 Fig. 23 and Fig. 24 form a flow chart show ing the disambiguation and address decoding 

processes performed by a computer; 

Fig. 25 illustrates a binary data matrix formed from a glyph lattice; 

Fig. 26 is a flowchart show ing an embodiment of correlation steps of Fig. 24; 

Fig. 27 illustrates why the correlations detemiine which w^ay the codes in every other row 
20 are shifting; 

Fig 28 show^s a method for expedited processing wiien successive captures are processed 
from an image capture device; 
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fMg. 29 is a block diagram of a user interface image capture system that may be used to 
capture a user-selected portion of a graphical user interface; 

Fig. 30 is a block diagram illustrating another embodiment of an image registration system 
consistent with the principles of the invention; and 
5 Fig. 3 1 is a block diagram showing a cutaway side view of the camera mouse. 

DETAILED DESCRIPTION OF THE INVENTION 
Reference will now be made in detail to embodiments of the invention, examples of which 
are illustrated in the accompanying drawings. Methods, apparatus, and articles of manufacture 
disclosed herein consistent with the principles of the invention register first information having 
1 0 embedded data with second information. 

Fig. 1 illustrates glyph marks and codes embodied in the glyph marks. Glyph marks are 
typically implemented as a fine pattern on a substrate, such as glyph marks 21 on substrate 24. 
Glyph marks are not easily resolved by the unaided human eye. Thus, glyph marks typically 
appear to the unaided eye as having a uniform gray scale appearance or texture, as illustrated by 
1 5 glyph marks 2 1 in Fig. 1 . 

Enlarged area 23 shows an area of glyph marks 21 . Glyph marks 21 are comprised of 
elongated slash-like marks, such as glyph 22, and are typically distributed evenly widthwise and 
lengthwise on a lattice of glyph center points to fomi a rectangular pattern of glyphs. Glyphs are 
usually tilted backward or forward, representing the binary values of "0" or " 1 respectively. For 
20 example, glyphs may be tilted at +45° or -45'' with respect to the longitudinal dimension of 
substrate 24. Using these binary properties, the glyph marks can be used to create a series of 
glyph marks representing O's and I's embodying a particular coding system. 
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The glyph marks of enlarged area 23 can be read by an image capture device. The 
captured image of glyph marks can then be decoded into O's and I's by a decoding device. 
Decoding the glyphs into O's and Ts creates a glyph code pattern 25. The O's and I's of glyph 
code pattern 25 can be further decoded in accordance with the particular coding system used to 
5 create glyph marks 21 . Additional processing might be necessary in the decoding stage to resolve 
ambiguities created by distorted or erased glyphs. 

Glyph marks can be implemented in many ways. Apparatus and methods consistent with 
the invention read and decode various types of glyph code implementations. For example, glyphs 
can be combined with graphics or may be used as halftones for creating images. 
10 Fig. 2 illustrates an embodiment of an image 210 combining graphics and glyphs 

consistent w ith the present invention, in this particular embodiment, the graphics comprise user 
interface icons. Each icon comprises a graphic overlaid on glyphs. The glyphs form an address 
carpet. The glyph address carpet establishes a unique address space of positions and orientations 
for the image by appropriate coding of the glyph values. 
15 Fig. 3 illustrates an enlarged view of a portion of image 210 illustrated in Fig. 2. More 

particularly, portion 212 illustrates the Lab.avi icon overlaying a portion of the address carpet, 
which unambiguously identifies the icon location and orientation. 

Fig. 4 illustrates an image of a pictorial comprising glyphtones consistent with the present 
invention. Glyphtones are halftone cells having area-modulated glyphs that can be used to create 
20 halftone images incorporating a glyph code. 

Fig. 5 illustrates an example of a portion of a glyph address carpet code with overlay tiled 
label code. The u and v glyph address codes comprise the address carpet, and the d codes 
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comprise the overlay tiled label code. In this embodiment, rows of u address codes are 
interleaved with rows of v codes. Each row of u address codes is offset by two positions so that a 
diagonal line running down and to the right at 45 cuts across identical u codes in every other 
row. Similarly, each row of v address codes is offset by two positions so that a diagonal line 
5 running down and to the left at 45 cuts across identical v codes in every other row. This 

property of the u and v address codes allows determination of the precise location within the code 
from a captured image of glyphs. 

Every sixth column of the address carpet is substituted by d codes, which comprise a label 
code. A label code may represent additional information, such as a page number of context. The 

10 label code in Fig. 5 is a four-bit label code, comprising bits do,, do2. ^03, and do4. The d codes 

repeat throughout the address carpet. The d codes substitute for u and v codes. For example, in 
the top row, v,^, V]g, v.,, and v,, are substituted for do;, do4, dn,, and do., respectively. In each row, 
the u and v codes in every sixth column are over written by corresponding d codes. In a captured 
portion of the address carpet, the d codes can be decoded to provide a label providing information. 

1 5 The d codes could represent, for example, a page number or context infomiation. Thus, if the 
portion of glyph address carpet illustrated in Fig. 5 were read, codes do^, do4, doi, and do. 
respectively represented 1, 1, 0, 0, the reordered d codes would form a code of doi, do2, do^, and 
do4, a code of 001 1, indicating page or context three (binary 00 11 =3). The proper logical identity 
of the code elements (i.e., addressing) is provided by the address code. 

20 From the captured portion of the glyph address carpet code having overlay tiled label code 

as illustrated in Fig. 5, orientation, location, and a label can be detennined. The u and v address 
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codes can be used lo dctemiinc position, as w ill be explained in greater detail below, and the d 
codes provide label infomiation. 

Apparatus and methods consistent with the invention read embedded data from a substrate, 
decode the embedded data to determine registration of the substrate relative to a reference in an 
5 apparatus, and develop and present human-sensible infomiation based on the registration 

information. In one embodiment, the human-sensible information is visual infomiation registered 
with the substrate. The human-sensible information may also comprise, for example, tactile, 
audible, or other sensory infomiation. 

Fig. 6 illustrates a system for reading an image having embedded data, decoding the 
10 embedded data in the image, and developing human-sensible infomiation based on the decoded 
embedded data. More particularly, image capture 470 reads substrate 468 to capture an image 
having embedded data, decoder 472 decodes the embedded data in the captured image, and 
infomiation generator 474 develops human-sensible information based on the decoded embedded 
data, and outputs the infomiation to infomiation output 476, which represents one or more 
1 5 infomiation output devices. The human-sensible infomiation may be visual mfomiation 

registered with substrate 468, and additionally or altematively may comprise other human- 
sensible information, such as tactile, audible, or other human-sensible information. 

Fig. 7 is a block diagram illustrating a logical configuration of elements in accordance 
with principles consistent with the invention. An image capture device 70 captures an image from 
20 a substrate 68. Substrate 68 has embedded data, such as glyphs embodied thereon. Image capture 
device 70 transfers the captured substrate image to a decoder 72 and an image generator 74. In 
one embodiment, the embedded data on substrate 68 comprises an address carpet and tiled label 
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code. Decoder 72 analy/cs the embedded data in the captured substrate image to determine 
information regarding the registration of the substrate, such as orientation, location, and label 
information. These results are transferred to image generator 74 for further processing. 

Image generator 74 processes the results from decoder 72 and the captured substrate image 
5 from image capture device 70. In one embodiment, image generator 74 rcceix es a context code, a 

location code, and a rotation code from decoder 72. The context code defines a particular context 
of substrate 68, such as a substrate page number. The location code defines the location on 
substrate 68 where the captured image was taken from. The rotation gives the orientation of the 
substrate relative to the image capture device. Using the context code and location code, image 

10 generator 74 accesses information related to the context code and location code. For example, if 
the context code represents substrate page iv, location code indicates location coordinates x^97 
and y=92, and rotation ^43"', image generator 74 retrieves coiresponding information and 
generates an image registered to the substrate. The rotation code enables registering the 
orientation of the generated image. 

1 5 In one embodiment, the retrieved infomiation includes two types of infomiation: image 

infonnation in the form of a bit map image representation of substrate page iv, and icon 
infomiation defining the locations and meaning of icons on substrate page iv. Using this 
infomiation, if the user obser\ es an icon in display 76 that is desired to be selected, the user can 
provide a selection signal. In response, a system (not shown) attached to the registration system 

20 of Fig. 7 can use the retrieved meaning of the icon to perfomi functions corresponding to the icon, 
as a conventional system would in response to user selection of an icon on a user interface in a 
conventional display. 
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The size of the retrieved information may vary. In one embodiment, image generator 74 
retrieves an image of substrate 68 that is the same si/e as the footprint of disphiy 76 and 
corresponds to the area of substrate 68 directly under the footprint of display 76. Because display 
76 is aligned with substrate 68, observer 78 looking at display 76 is given the illusion of looking 
5 directly onto substrate 68, Image generator 74 may also add information to the image, or alter the 
retrieved image before sending it to display 76. 

The image sent to display 76 may be generated by image generator 74 in many ways. For 
example, image generator 74 may merely pass on the image captured by image capture 70, or a 
representation of the image captured by image capture 70. For example, a bitmap representation 

1 0 of the entire substrate 68 could be stored locally in image generator 74 or on a remote device, 

such as a device on a network. In one embodunent, in response to receiving codes from decoder 
72, image generator 74 retrieves an area corresponding to the codes from the bitmap 
representation, and forwards the area representation to display 76 for display to a user. The area 
representation retrieved by image generator 74 may be the same size as the image captured by 

1 5 image capture 70, or may be an extended view, including not only a representation of the captured 
area, but also a representation of an area outside the captured area. The extended view approach 
only requires image capture 70 to be as large as is necessary to capture an image from substrate 68 
that is large enough for the codes to be derived, yet still provides a perception to the user of seeing 
a larger area. 

20 Fig. 8 is a block diagram illustrating an embodiment of a system consistent with the 

principles of the invention. A substrate 89 having embedded data thereon is positioned below a 
semitransparent mirror 82. An image from substrate 89 is captured by an image capture device 
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80. Image capture device 80 sends the captured image to a decoder 88, which decodes the image 
and determines codes from the captured image. Decoder 88 sends the codes to an image generator 
84. hnage generator 84 processes the codes, creates and/or retrieves image information based on 
the codes, and sends the image infomiation to semitransparent mirror 82. 
5 An observer 86 looking down onto semitransparent mirror 82 sees the image generated by 

image generator 84 overlaid on the image from substrate 89. \n this way, the overlaid information 
can be dynamically updated and registered w ith information on substrate 89 based on the decoded 
image captured by image capture device 80. 

hi an alternative embodiment, image capture 80 receives the substrate image reflected 
1 0 from semitransparent mirror 82. 

Fig. 9 is another embodiment of a system built in accordance w ith the principles of the 
present invention. An image from substrate 98 is reflected off semitransparent mirror 96 onto 
projection surface 102. The image from substrate 98 is also captured by an image capture device 
90. hiiagc capture device 90 transfers the image to a decoder 92, which processes the image to 
15 decode registration infomiation and to detemiine whether further infomiation should be 

generated. Based on the detemiination, decoder 92 passes signals to image generator 94, directing 
image generator 94 to generate an image. Image generator 94 generates an image, which is 
projected onto projection surface 102. Observer 100 viewing projection surface 102 sees the 
image from substrate 98 overlaid and registered with the image generated by image generator 94. 
20 The system may also include illumination source 104 for illuminating substrate 98. 

In each of the systems of Fig. 6, Fig. 7, Fig. 8, and Fig 9, the elements may send 
information to and receive infomiation from network devices. This allows the elements to 

1 2 Attorney Docket No.: ()7447.()04?-0000() 




inleracl with devices on a network. Vov example, programs and data may be sent to the elements 
from network devices, and the devices may send information to the devices on networks. 

Fig. 10 is a diagram illustrating registration of information consistent with the principles 
of the invention. The process may be carried out by the apparatus discussed above. Substrate 364 
5 has embedded code embodied thereon, and may have images, such as a triangle. The embedded 
code embodies a code system from w hich x,y positions on substrate 364 can be determined. 

An image capture device captures a portion of substrate 364, to thereby capture an image 
of a portion of the embedded code embodied thereon. The embedded code is decoded to 
determine an x,y location within the embedded code, and the orientation of substrate 364, 
10 represented by the crosshair arrow on substrate 364. A label code may also be decoded from the 
captured embedded code. 

Based on the label code, image information 366 is retrieved from storage. The x,y 
location information and orientation information decoded from the embedded code embodied on 
substrate 364 are then used to register image infonnation 366 with substrate 364. These may be 
1 5 used to form a composite image 368. 

Fig. 1 1 is a block diagram illustrating an embodiment of a lens apparatus consistent with 
the principles of the invention. Lens apparatus 328 is comprised of lens view port 334, which is 
supported by support arm 330. A viewer looking down through lens viewport 334 observes 
substrate 332, which has embedded code embodied thereon. A camera (not shown) captures an 
20 image of substrate 332. The image is sent to a computer (not showqi), which decodes the x,y 
location of substrate 332 appearing under lens \ iewport 334, the orientation of substrate 332 
under lens viewport 334, and the label code, if any, in the embedded code on substrate 332. 
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Based on ihc label, x,y location and orienlalion of substrate 332, the computer generates overlay 
image information which is displayed in lens viewport 334 in such a way that the generated image 
information is registered with substrate 332. The registered overlay image is projected by an 
image generator (not show^n). 
5 Fig. ] 2 is a cutaw ay side view of the lens apparatus shown in Fig. 1 1 . Lens apparatus 328 

further comprises camera 392, display 394, lamp 396, display controller 398, computer 400 and 
semitransparent mirror 402. Lamp 396 illuminates substrate 332 (not shown). Camera 392, 
which corresponds to image capture devices 70, 80, 90 illustrated in Fig. 7, Fig. 8, and Fig. 9, 
respectively, captures an image of the substrate, and transmits the image to computer 400. 

10 Computer 400 performs the function of decoders 72, 82, 92 illustrated in Fig. 7, Fig. 8, and Fig. 9, 
respectively. Computer 400, in combination w ith display controller 398 and display 394, 
performs a function most similar to image generator 84 illustrated in Fig. 8 because the generated 
image is reflected off semitransparent mirror 402. 

Computer 400 decodes the embedded data in the captured image to determine the x,y 

1 5 location of the captured image, w hich represents the location of the area on substrate appearing 
under lens viewport 334. 

Computer 400 also decodes the embedded data in the captured image to determine the 
orientation of substrate 332 under lens viewport 334, and the label code, if any, in the embedded 
code of the captured image. From this infomiation, computer 400 generates the overlay image 

20 information, which is sent to display controller 398. Display controller 398 sends the overlay 

image information to display 394. Display 394 generates an overlay image based on the overlay 
image infonnation from display controller 398. Observer 390 looking through viewport 334 sees 
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substrate 332 through scmitransparcnt mirror 402 overlaid with the overlay image information 
generated by image generator 394. 

Fig. 13 illustrates an example of a substrate, an overlay image, and the substrate overlaid 
with the overlay image as seen through the lens viewport illustrated in Fig. 1 1 and Fig. 12. The 
5 substrate 480 is comprised of images of North and South America and embedded data. In one 
embodiment, the substrate is covered entirely with embedded data. A user places substrate 480 
under lens viewport 334 and camera 392 captures the image appearing under lens viewport 334 
and transmits the image to computer 400. Computer 400 decodes the embedded data in the 
captured image from substrate 480 to determine the x,y location of the captured image, which 
1 0 represents the location of the area on substrate appearing under lens viewport 334. Computer 400 
also decodes the embedded data in the captured image to determine the orientation of substrate 
480 under lens viewport 334, and the label code, if any, in the embedded code of the captured 
image. 

From this information, computer 400 generates overlay image information 482, which is 
1 5 sent to display controller 398. Display controller 398 sends overlay image information 482 to 
display 394. Display 394 generates overlay image information 482, which is reflected off 
semitransparent mirror 402 through lens viewport 334. Observer 390 looking through viewport 
334 sees substrate 332 through semitransparent mirror 402 overlaid with overlay image 
information 482 generated by image generator 394. In 1 3, the user sees borders overlaid on North 
20 America. 

Fig. 14 illustrates another example of a substrate, an overlay image, and the substrate 
overlaid with the overlay image as seen through the lens viewport illustrated in Fig. 1 1 and Fig. 
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12. More particularly. Fig. 14 illuslralcs how Ihc system responds when the user moves substrate 
430 under lens viewport 334. in this example, substrate 430 comprises the lowercase letters "c 1" 
and "world." Substrate 430 also includes embedded data embodied thereon (not shown). When 
the user mo\'cs substrate 430 so that letter "e" is under lens viewport 334, camera 400 captures an 
5 image of the substrate area under lens viewport 334. Computer 400 decodes the embedded data 
in the captured image from substrate 430 to determine the x,y location of the captured image, 
which represents the location of the area on substrate appearing under lens viewport 334. 
Computer 400 also decodes the embedded data in the captured image to determine the orientation 
of substrate 430 under lens viewport 334, and the label code, if any, in the embedded code of the 

1 0 captured image. 

From this information, computer 400 generates overlay image infomiation "H 1," which is 
sent to display controller 398 and reflected off semitransparent mirror 402. A user looking 
through lens viewport 334 sees the "e" overlaid with overlay image information "H 1," as 
illustrated in the upper right of 14. When the user moves substrate 430 so that "1" appears under 

1 5 lens viewport 334, camera 392 captures an image of the new area under lens viewport 334. 

Computer 400 decodes the embedded data in the new area, and generates an image of "o," which 
is sent to display controller 398. A user looking through lens viewport 334 sees the "1" overlaid 
with overlay image information "o," as illustrated in the lower right of 14. Thus, as the user 
moves substrate 430, the overlay image infomiation is dynamically modified to appear in lens 

20 viewport 334. More particularly, the overlay image infonnation is dynamically modified to 
maintain registration w^ith the area of substrate 430 appearing under lens viewport 334. 



1 6 Attorney Docket No.: 07447.0043-00000 



Each of the systems described w ith respect to Fig. 7, Fig. 8, Fig. 9, and Fig. 12, register 
image information with a substrate. Registering requires decoding the embedded data on the 
substrate to determine the content of the information to be registered and the orientation of the 
information relative to the substrate. Therefore, the decoding and registration techniques 
5 described herein can be apphed in any systen^ consistent with the principles of the invention. 

Registering the overlay image with the substrate requires a precise detennination of the 
orientation of the substrate w ith respect to the image capture device. To determine the orientation 
angle of the substrate relative to the image capture device, computer 400 resolves the angle 
between 0 ' and 360 This is done in two steps. First, the quadrant offset angle (between -45° 

10 and +45 " ) from the lattice axis to the nearest quadrant axis is detemiined. Second, the quadrant 
angle of 0", 90°, 180" or 270 ' is determined. The sum of the quadrant offset angle and the 
quadrant angle is the orientation angle. 

Fig. 15 is a flowchart showing the processing perfomied by computer 400 to register an 
image. Computer 400 first captures an image of the substrate, including a lattice of glyphs (step 

1 5 1 30), and then determines the quadrant offset angle of the lattice of glyphs betw^een -45 ° and 

+45 ' relative to the image capture device (step 132). Computer 400 then decodes the glyphs in 
the image into data composed of O's and I's (step 134). From the decoded data, computer 400 
detemiines the quadrant angle of 0 , 90 \ 180 , or 270", and then correlates the data from step 
1 33 with the address carpet code used to create the address carpet to determine the x,y location of 

20 the captured image (step 1 36). Computer 400 sums the quadrant offset angle and the quadrant 
angle to determine the orientation angle between the lattice code and the image capture device. 
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Computer 400 also delcnnincs a context code from the decoded data (step 138). Using the 
context code, computer 400 generates a context-specific overlay (step 140), and applies a 
context-specific x,y, scale and rotation offset values to the overlay (step 142). Finally, computer 
400 applies the calculated x,y and rotation values to the overlay (step 144), and generates image 
5 information for display of the image (step 146). 

In an another embodiment, the glyphs do not include a context label. In this embodiment, 
step 138 is not performed and retrieval of image information is based only on location code. 

Fig. 16 illustrates graphically the process of creating an image of superimposed 
neighborhood images in a glyph lattice image, in accordance with the flowchart of Fig. 17. More 
10 particularly, in the embodiment shown in Fig. 16, representative neighborhood images 204, 206, 
208, and 210 of glyph lattice image 200 arc superimposed to fomi the composite image 202 of 
superimposed neighborhood images. From composite image 202, a quadrant angle can be 
determined, as discussed in detail below. 

Fig. 1 7 is a flowchart showing the processing performed by computer 400 to create a 
1 5 composite lattice image for use in detemiining the quadrant offset angle. Computer 400 first 

selects a seed pixel from the image captured by camera 392 (step 1 62). In one embodiment, the 
selection of seed pixels begins at captured image coordinate 0,0. Computer 400 finds a local 
minimum in the vicinity of the seed pixel, indicating the presence of a glyph (step 164). 
Computer 400 then finds the centroid of this glyph (step 166), and determines whether the glyph 
20 has sufficient contrast with its immediate surround to be reliably decoded (step 168). Glyph 
centroids are discussed in greater detail with respect to Fig. 19. Computer 400 determines 
whether there have already been too many sampling attempts (step 1 74). If too many sampling 
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allempts, the process (ails (step 172). If not loo many, processing continues at step 162 by 
selecting the next seed pixel for analysis at a particular x and y interval from the previously 
analyzed seed pixel (step 162). The particular x and y interval is based on the height and width of 
the composite image. 

If there is sufficient contrast (step 168), computer 400, using the glyph ccntroid as the 
origin, adds a subsamplc of size w x h of the captured image to the composite image (step 176). 
Computer 400 then detemiines whether enough samples have been taken (step 1 78). If the limit is 
not exceeded, computer 400 determines whether there have been too many sampling attempts 
(step 170). 

If the sampling attempt limiVhas been exceeded, computer 400 divides the values of the 
composite image by the number of successful samples to detemiine an average of the values of 
pixels in the subsamplc regions (step 1 80). From the resulting composite image, computer 400 
determines the quadrant offset angle (step 1 82), as illustrated in 1 8. The preciseness of the 
quadrant angle is proportional the size of the composite image. Put another w ay, a large 
composite image will provide a more precise quadrant angle than a small composite image. 

Fig. 1 8 is a flowchart showing the processing perfonned by computer 400 to determine a 
quadrant offset angle using a composite lattice image generated in accordance w ith the flowchart 
of Fig. 17. Computer 400 first finds the darkest pixel along an arc between zero and 90 degrees at 
a distance from the origin equal to the glyph pitch, the distance between adjacent glyphs on the 
lattice of glyphs (step 230), and then finds the ccntroid of the shape containing this pixel (step 
232). Once the ccntroid is found, computer 400 assumes that the lattice axis passes through this 
ccntroid and the origin, so it then estimates the approximate location of the next minimum along 
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Ihc lattice axis based on the centroid position and the glyph pilch (step 234). Using this estimate, 
computer 400 finds the local minimum around the estimated location (step 236), and finds the 
centroid of the shape containing that minimum (step 238). Based on the size of composite image, 
computer 400 determines whether the last possible minimum along the lattice axis been found 
(step 240). If the last possible minimum has not been found, processing continues at step 234. If 
the last possible minimum has been found, computer 400 fits a straight line, referred to as the axis 
line, from the origin through the centroids (step 242). Computer 400 then determines the angle of 
the axis line, between 0" and 90' (step 244), and this angle is then offset to fall between -45 
degrees and +45 degrees by subtracting 45'' (step 246). 

Fig. 19 illustrates graphically one way to find a glyph centroid consistent with the 
principles of the invention. The Y coordinate of the centroid is found by: 




The X coordinate of the centroid is found by: 
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Cenlroids are used in the present invention in two ways: to align the subimages, and in the case of 
angle finding the ccntroid is used to more acciiralcly establish the axis of the lallice, thus 
providing the angle. 

Fig. 20 and Fig. 21 illustrate the process of analyzing the composite image, as illustrated 
5 in Fig. 1 6, to accurately determine the rotation angle of the composite image. The quadrant offset 
angle of the composite image represents an accurate determination of the orientation of the glyph 
lattice in the original captured image, subject to determination of the proper angular quadrant. 
Combining the quadrant offset angle and the angular quadrant provides a complete determination 
of the relative angle of the image capture device and the substrate. Fig. 20 illustrates one example 
1 0 where the quadrant offset angle is 0°, and Fig. 21 illustrates another example where the quadrant 
offset angle is the finite angle theta. 
. \ Apparatus and methods consistent with the i/ivention use address carpet codes and 

associated processes similar to those shown ii>t}.S. Patent Application No. , entitled 

GLYPH ADDRESS CARPET METHQZJS AND APPARATUS FOR PROVIDING LOCATION 

/ 

1 5 INFORMATION IN A MULTIDpd^ENSIONAL ADDRESS SPACE (Attorney Docket No. 

/ 

07447.0010-00000), filed August 31, 1998, which is hereby incorporated by reference. 

* 

Fig. 22 illustrates an embodiment of the address codes encoded in a portion of a glyph 
address carpet similar to the u and v codes described above. The addresses are encoded by 
alternating rows of "A" address code sequences and "B" address code sequences. The position 
20 along each sequence in each row should be unambiguously determinable from a predetermined 
length subsequence. For example, an N-bit shift register maximal length code can be uniquely 
detemiined in position from an N-bit subsequence. Each address code sequence is a fifteen bit 
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sequence, with the A sequence indexed running left to right, and the B sequence indexed running 
in a reverse direction, right to left. Each row of A code sequences is olTscl by two glyph positions 
relative to the previous and next row of A addresses. Similarly, each row^ of B code sequences is 
offset in the opposite direction by two positions. Thus, the encoding scheme has two key 
5 characteristics: parallel rows including two sets of one-dimensional unique address codes and 

relative offsets among members of the two sets so that the offset between each pair from the two 
sets is unique. This establishes two-dimensional unique address locations. 

Computer 400 decodes address information encoded in the glyphs by analyzing the 
captured image area in two steps. Ideally, in the systems shown and described with respect to Fig. 

10 7, Fig. 8, Fig. 9, and Fig. 12, image capture devices 70, 80, 90, and 392, respectively, capture an 
area of a substrate that is angularly aligned as shown in the pattern of bits shown in 22. In reality, 
how ever, the substrate and image capture device may not be aligned to one another. Thus, the 
relative angle between the two could be oriented anywhere from 0" to 360". Therefore, computer 
400 must first determine the orientation of the image as part of decoding and interpreting the 

1 5 address infonnation. 

The orientation of the image is determined by analyzing the captured image. This process 
is called disambiguation. One method of disambiguation is described in U.S. Patent No. 
5,521,372 to Hecht et al. expressly incorporated herein by reference. After detemiining the proper 
orientation of the image, computer 400 decodes the address of the selected location in the address 

20 carpet. Disambiguation and address decoding processes perfomied by computer 400 and 
consistent w ith the present in\'cntion will now be described in greater detail. 
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Fig. 23 and Fig. 24 fonn a flow chart showing exemplary disambiguation and address 
decoding processes performed by computer 400 on the captured image area. Computer 400 
begins the disambiguation process by image processing the captured portion of the address carpet 
to fmd a glyph seed. A glyph seed is a first-identified glyph having readable glyphs around it. 
5 Once a glyph seed has been found, the glyph lattice can be determined by processing neighbors of 
the glyph seed. (See Appendices A and B). The glyphs are then decoded as 1 's or O's, which are 
filled into a binary data matrix having rows and columns corresponding to the glyph lattice rows. 
The orientation may still be ambiguous with respect to 90'' and 180" rotations. 

Fig. 25 illustrates a binary data matrix (BDM) 2310 formed from a glyph lattice captured 
10 by the image capture device. Locations in the BDM correspond to locations in the glyph lattice, 
and thus the size of the BDM corresponds closely to the size of the glyph lattice. 

Each location of the glyph lattice is analyzed to determine which value should be placed in 
the corresponding location of the BDM. Initially, the BDM is filled with a value, for example (f), 
which indicates that no attempt has been made to read the glyph. Once the glyph corresponding 
15 to a particular location has been analyzed, (J) is replaced by a value indicating the result of the 
glyph analysis. 

In Fig. 25, a B indicates a border location, an X indicates that no interpretable glyph was 
found at the corresponding location of the glyph lattice, an E indicates a glyph at the edge of the 
captured image portion, a 0 indicates a back slash glyph, a 1 indicates a forward slash glyph, and 
20 d indicates a label code. The area of the matrix corresponding to the captured image is filled with 
O's and Ts, the edge is bounded by E's. and the X's correspond to locations that have no readable 
glyphs. Although the BDM will generally have a similar overall pattern to those of Fig. 25, the 
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\aliics will often not be as evenly distributed as shown m Fig. 25. For example, a glyph location 
within the captured image area might result in an X if the glyph has been obliterated. Several 
\alucs have been drawn with circles and squares around them to illustrate the two separate code 
sequences that are staggered in opposite directions. 
5 The image capture de\ ice might be oriented relative to the substrate at any angle. 

Therefore, the captured image could be oriented at any angle. Thus, even though a BDM of O's 
and Ts is derived from the captured image, it is uncertain whether the BDM is oriented at 0" (i.e., 
correctly oriented), 90'\ 1 80", or 270" relative to the original code pattern in the glyph address 
carpet from which the image was captured. Until the orientation of the captured image is 
1 0 determined, it is not possible to derive the correct address code from the BDM. The orientation 
could be provided by auxiliary information such as physical system constraints. How^ever, the 
orientation can be uniquely determined directly from the address codes. 

After converting the glyphs to O's and Fs, a reference glyph location is selected (step 

221 1) . This location may be chosen in a variety of ways, but is typically a location which 

1 5 represents the selection. For example, the reference glyph location could be at the center of the 
BDM. 

After the image has been converted to a BDM, it is processed by computer 400 (step 

2212) . The original BDM developed from the captured image is referred to as BDMl. Computer 
400 makes a copy of BDMl and rotates the copy clockwise 90" to form a second binary data 

20 matrix, BDM2 (step 2214). By rotating BDMl by 90", the rows of BDMl become the columns of 
BDM2, and the columns of BDMl become the rows of BDM2. Additionally, all bit values in 
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BDM2 arc (lipped from 0 to 1, and 1 lo 0, because a 45*' slash glyph rotated 90" appears as the 
opposite state of the non-rotated glyph. 

Computer 400 then performs a correlation separately on the odd and e\cn rows of BDMl 
(step 2216) to determine whether code in the rows are staggered forward or backward. The 
5 correlation is also performed for the odd and even rows of BDM2 (step 221 8). The correlation is 
performed over all the rows of each EDM, and results in correlation value CI for BDMl and 
correlation value C2 for BDM2. 

Fig. 26 is a flowchart showing an embodiment of correlation steps 2216 and 2218 of Fig. 
24. The process determines a correlation value for every other line of a BDM along diagonals in 
10 each direction, and sums the row correlation values to form a final correlation value for the odd or 
even rows. The process is performed on the odd rows of BDM 1 to form correlation value CI odd 
for BDMK the even rows of BDMl to form correlation value CIhmn for BDMl, the odd rows of 
BDM2 to form correlation value C2(ji)i) for BDM2, the even rows of BDM2 to form correlation 
value C2, VI N for BDM2. The BDM that is oriented at 0" or 18(r w ill have a larger C^m + C, vi n 
15 than the other BDM. 

Computer 400 first inputs the BDM (step 2410), and then copies every other row to a 
temporary matrix (step 2412). Identical processes are then performed for diagonals to the right 
and to the left. Steps 2414, 2416, 2418, 2420, 2422 and 2424 process the diagonals to the right. 
For example, in Fig. 27 the steps correlate along the diagonals moving from the upper left to 
20 lower right. First, row count N and correlation value C RIGHT are each initialized to zero (step 
2414). Row N is shifted two places to the right, and correlated with the next row (step 2416). 
C_N is then set to this value (step 2418). C RIGHT is then set to C RIGHT + C N (step 2420), 
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and N is incremented (step 2422). If row count N Nniax, where N is the number of odd or 
even rows in the BDM, then the process proceeds to step 2426. If N is not greater than or equal to 
Nmax, the process continues at step 2416. Thus, after the process has correlated each adjacent 
row, the correlation value C RIGHT indicates the strength of the correlation along the diagonals 
5 to the right. 

The steps on the right of Fig. 26 are similar to steps 2414, 2416, 2418, 2420, 2422 and 
2424, but process diagonals running from the upper right to lower left to develop correlation value 
C_LEFT. After correlating the right and left diagonals to detemiine C RIGHT and C LEFT, a 
final correlation value C is determined by subtracting C LEFT from C RIGHT. For example, if 

10 odd rows for BDMl are processed, the C value becomes CI odd for BDML 

The process steps of Fig. 26 are performed for the odd and even rows of BDMl and the 
odd and even rows of BDM2. From this infonnation, the correlation value CI for BDMl is set to 
CliviN CI ODD (as detemiined by Fig. 26 for the row^s of BDMl), and the correlation value C2 
for BDM2 is set to C2, yj ^ + C2o[,d (as determined by Fig. 26 for the rows of BDMl). 

1 5 Fig. 27 illustrates why the correlations determine which way the codes in every other row 

are shifting. For example, as indicated by the circled Als along the diagonals running to the right, 
the codes along the diagonal startmg at Al in the first position of the second row should have the 
same value in every other row along the diagonal, except for erasures and errors. Similarly, as 
indicated by the boxed Bis, the codes along the diagonal starting at Bl in the upper right comer 

20 should have the same value in every other row along the diagonal, except for erasures or errors. 

This is true for each value along the diagonal in the odd rows running respectively from B2, B3, 
... Bl 5 in the top row. Thus, the strong correlations along the diagonals running down and to the 
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left on the odd rows, and the strong correlations along the diagonals running down and to the right 
on the even rows indicate that the codes in the even rows are shifting to the right, and the codes in 
the odd rows are shifting to the left. 

For each BDM, therefore, four correlation values arc developed: 1 ) odd rows, right to left, 
5 2) odd rows, left to right, 3) even rows, right to left and 4) even rows, left to right. From these 
correlation values, the strongest correlation value for the even rows, and strongest correlation 
value for the odd rows is chosen, and these become Ci vj \ and Cood for that BDM (steps 2216 and 
22 1 8). CiM x and Conn are then added to form a final C correlation value for that BDM. As 
discussed above with respect to step 2220, the BDM with the strongest correlation value is the 

10 BDM that is oriented at either 0" or 180'' because of the relative orientation of the codes in the odd 
and even row s. Thus, two aspects of the chosen BDM are now^ established; w hich direction every 
other line of codes is staggered, and that the BDM is oriented horizontally, at either 0" or 180*\ 
Another correlation process, step 2230 is performed to determine which direction the code in each 
line runs (as opposed to which way the code is staggered). 

1 5 The codes in the odd lines are staggered in one direction, and the codes in the even lines 

are staggered in the other. This staggering property of the code, in conjunction with knowing the 
respective codes that run in the odd lines and even lines, allows determination of the proper 0"" 
orientation of the BDM. 

Returning to Fig. 23, if CI is greater than C2 (step 2220), then BDMl is selected for 

20 further processing. CI being greater than C2 indicates that the one-dimensional codes of BDMl 
are most strongly correlated and are, therefore, oriented at either 0" or 1 80" (step 2222). If C2 is 
greater than CI, then BDM2 is selected for further processing, because the higher correlation 

27 Attorney Docket No.: 07447. 0043-00000 



indicates that BDM2 is oriented at either 0" or 180'* (step 2224). Thus, the correct BDM has been 
found. Before determining the address location of the captured image, however, computer 400 
must first dctemiinc whether the selected BDM is at if (i.e., oriented correctly), or rotated by 
180^ 

5 Fig. 24 is a flowchart showing the steps of how computer 400 determines the address of 

the captured area of the glyph carpet. Preferably, bit positions along a diagonal in the BDM, 
when the BDM is oriented at 0", have the same value at every other row. The image capture 
process and interference from the visual indicia and substitution label codes d (as show^n in Fig. 
25), how ever, may result in errors and erasures in the BDM data. To reduce the impact of these 

1 0 errors and erasures, computer 400 performs a majority vote along each diagonal of the odd rows 
in the direction the odd row^s arc staggered, and repeats the majority vote process for even rows 
along the each diagonal in the direction the even rows are staggered (step 2225). This results in a 
first code sequence for the odd rows and a second code sequence for the even rows. To the extent 
the majority vote correctly detennines each bit position, the first and second code sequences 

15 should match a subsequence of the original pseudo noise address sequence respectively 
corresponding to the odd or even set of rows. 

Computer 400 then retrieves the original pseudo noise address code (Get Code 1 ) for rows 
staggered forward (step 2226), and retrieves the original pseudo noise address code for (Get Code 
2) for rows staggered backward (step 2228). Using the original pseudo noise address code for 

20 each code set A and B, and the code from the majority voting, computer 400 performs four cross 
correlations (step 2230) to establishes the best match of the glyph sequence with the PN sequence 
location for the odd and even rows. 
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More particularly, two adjacent rows of the BDM closest to the reference clement chosen 
in step 221 1 are correlated with the respective complete PN sequences that created the original 
address carpet. The PN sequences could be identical. A forward and backward correlation is 
performed for each row. The four correlations develop four pairs of peak correlation and position 
5 values: 

1) PI, VI, respectively representing the peak correlation value and corresponding position 
for Code 1 correlated backw^ards with the complete PN sequence; 

2) Ql, Ul, respectively representing the peak correlation value and corresponding 
position for Code 2 correlated forwards with the complete PN sequence; 

10 3) P2, V2, respectively representing the peak correlation value and corresponding position 

for Code 1 correlated forwards with the complete PN sequence; and 

4) Q2, U2, respectively representing the peak correlation value and corresponding 
position for Code 2 correlated backwards with the complete PN sequence. 

The U, and V, position values, where i - 1 or 2, corresponding to the peak magnitudes are 
1 5 used to determine the X and Y values corresponding to the reference element chosen in step 221 1 . 
Thus, if (PI + Ql) > (P2 + Q2) (step 2232), then Ul and VI are used to calculate the X, Y 
position of the reference glyph location chosen in step 221 1 (step 2236). If (PI + Ql) <^ (P2 + 
Q2) (step 2232), then U2 and V2 are used to calculate the X, Y position of the reference glyph 
location chosen in step 221 1 (step 2234). The address information is determined in accordance 
20 with the following equations: 

X = ( V, - U, + length of full code)/2 
Y = (V, + U, - length of full code)/2 
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The calculated X, Y position is then relumed (step 2238). Note thai diagonals correspond 
to constant values of U and V, respectively, while rows and columns correspond to constant X 
and Y. Note also that U and V could be used directly as address parameters. 

Thus, an X, Y value associated with the reference point chosen in step 221 1 has been 
5 determined. Using this information, computer 400 associates the X, Y coordinates with a logical 
reference, or a combination of a logical reference and a control signal (e.g., a button click), with a 
particular operation to be perfonned. For example, the X, Y coordinates could be used as an 
index into a table of operations perfomiable by computer 400 or other device under direction of 
computer 400. The X, Y coordinates could be associated with a file open command that opens a 
1 0 fde associated with an icon located near the X, Y coordinates in the address space. Virtually any 
operation that can be performed by computer 400 could be associated with a particular X, Y 
coordinate, or range of X, Y coordinates. 

Fig. 28 show^s a method for expedited processing when successive image captures are 
processed from an image capture device when there is relative motion between the image capture 
1 5 device and the substrate at a velocity less than or equal to a predetermined value V. Initially, a 
first image is captured, and the entire address space is searched in accordance with the teachings 
above to detenriine the location coordinates of location #1 . In the example shown in Fig. 28, 
location #1 has coordinates u,^,, v... With motion velocity less than V, and time T betw^een 
captures, the next capture is constrained to be w ithin a circle of radius R=VT around the 
20 coordinales of location #1 . Therefore, the index range used in searching for a correlation 

maximum indicating location #2 can be limited to the index range u„v, of locations within the 
circle of radius R around the coordinates of location #F If a suitable correlation peak is not 

30 Attorney Docket No.: 07447.0043-00000 



found, processing returns to searching ihc entire address space. Not finding a suitable correlation 
peak might indicate that motion has jumped outside the circle or the substrate has been changed. 

The method expedites processing of information extraction and can enable applications 
such as real-time frame-to- frame image capture. This can be used for smoothed motion display 
5 and for gesture motion capture. 

Fig. 29 is a block diagram of a user interface image capture system that may be used to 
capture a user-selected portion of a graphical user interface, such as glyph address carpet 1732, 
and decode the glyphs in the captured portion. In one embodiment, computer system 1712 is a 
general purpose computer system, such as a conventional personal computer or laptop computer, 
1 0 that includes main memory 1716, read only memory (ROM) 1718, storage device 1 720, processor 
1 722, and communication interface 1 724, all interconnected by bus 1 726. Bus 1 726 also connects 
to registered image system 1 754, cursor control 1714, and frame capture 1 728. 

Registered image system 1 754 may be comprised of a system embodying principles as 
shown and described with respect to Fig. 7, Fig. 8, Fig. 9, and Fig. 12, or other registered image 
1 5 system consistent with the principles of the invention. For example, registration system 328 of 
Fig. 12 may be implemented w ith the system of Fig. 29 by using computer system 1712 as 
computer 400 in Fig. 12. Frame capture 1728 receives a captured image from registered image 
system 1 754 and provides the captured image to processor 1 722. Processor 1 722 decodes 
embedded data in the image to determine information, such as x,y location, label, and orientation 
20 information as described above, and sends image infonnation based on the decoded information 
to registered image system 1 754. Registered image system 1 754 generates an image registered 
with the substrate based on the image infomiation from processor 1722. 
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Registered image system 1 754 may recei\ e information from the other devices illustrated 
in Fig. 29, and may transmit information to the devices. This allows other devices, for example 
on a network, to interact with registered image system 1 754. 

Camera pen 1710 and mouse 1 752 can provide control signals to processor 1722 in 
5 conjunction wnth the operation of registered image system 1 754. Additionally, user interface 

substrate 1732 may be used as the substrate from wiiich registered image system 1754 captures an 
image. 

Camera pen 1 71 0 is connected to frame capture 1 728 and mouse 1 752 and acts as an 
auxiliary pointing device. Camera pen 1710 transmits image information to frame capture 1728. 

10 In one embodiment, button 1714 of camera pen 1710 is wired to mouse 1752 so that when a user 
presses button 1714 a signal travels through the circuitry of mouse 1752 to cursor control 1714. 
The signal causes processor 1722 to run a program that directs frame capture 1 728 to capture the 
image from camera pen 1710. In another embodiment, both the image line and signal line from 
camera pen 1710 are input directly into frame capture 1728. The lines between camera pen 1710 

1 5 and computer 1712 can be wired in any way that provides capture of the image from camera pen 
1710. 

The user makes a selection by placing camera pen 1 710 on or near visual indicia on glyph 
address carpet 1 732, and pressmg button 1714. Pressing button 1714 causes camera pen 1 7 1 0 to 
capture the portion of the address carpet 1 732 under the tip of camera pen 1710, and transmit the 
20 image to computer 1712, via frame capture 1728, for analysis. The button 1 714, or multiple 
buttons, can be used for additional signaling, as in a double click, hold dow n. 
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C'ompuler 1712 is also connected to local area network (LAN) 1 734 for communicating 
with devices connected to LAN 1 734. For example, LAN 1 734 may be connected to a computer 
1 740, a printer 1 736, and a mass storage 1 738. LAN 1 734 may be also connected to a gateway 
1 742 for connecting to another network 1 744. Network 1 744 may be connected to a computer 
5 1 746, a printer 1 748, and a database 1 750. By connecting computer 1 7 1 2 to LAN 1 734 and 

network 1744, computer 1712 can perform operations using the devices connected to these 
networks. For example, a document selected for printing by camera pen 1 7 1 0 or mouse 1 752 can 
be printed on printer 1 736 or printer 1748. Similarly, a user may use computer 1 712 to request a 
file on database 1750. 

1 0 In one embodiment, main memory 1 71 6 is a random access memory (RAM) or a dynamic 

storage device that stores instructions executed by processor 1 722. Main memory 1716 may also 
store infomiation used in executing instructions. ROM 1718 is used for storing static information 
and instructions used by processor 1 722. Storage device 1 720, such as a magnetic or optical disk, 
also stores instructions and data used in the operation of computer system 1712. 

15 Display 1730 may be a CRT or other type of display device. Cursor control 1714 controls 

cursor movement on display 1 730. Cursor control 1 714 may be, for example, a mouse, a trackball 
or cursor direction keys. 

The system shown in Fig. 29 can be used to implement the glyph address carpet capture 
and translation system described herein. The apparatus and methods described herein may be 

20 implemented by computer system 1712 using hardware, software, or a combination of hardware 
and software. For example, the apparatus and methods described herein may be implemented as a 
program in any one or more of main memory 1716, ROM 1718, or storage device 1720. In one 
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embodiment, processor 1722 executes programs which analyze captured portions of a glyph 
address carpet to detemiine address information encoded in the glyphs. 

Such programs may be read into main memory 1716 from another computer-readable 
medium, such as storage device 1 720. Execution of sequences of instructions contained in main 
5 memory 1716 causes processor 1 722 to perform the process steps consistent with the present 
invention described herein. Execution of sequences of instructions contained in main memory 
1716 also causes processor to implement apparatus elements that perform the process steps. 
Hard-wired circuitry may be used in place of or in combination w ith software instructions to 
implement the invention. Thus, embodiments of the invention are not limited to any specific 

10 combination of hardware circuitry and softw are. 

The term "computer-readable medium" as used herein refers to any medium that 
participates in providing instructions to processor 1 722 for execution. Such a medium may take 
many fomis, including but not limited to, non-volatile memory media, volatile memory media, 
and transmission media. Non-volatile memory media includes, for example, optical or magnetic 

15 disks, such as storage device 1 720. Volatile memory media includes RAM, such as main memory 
1716. Transmission media includes coaxial cables, copper wire and fiber optics, including the 
wires that comprise bus 1 726. Transmission media can also take the form of acoustic or light 
waves, such as those generated during radiowave and infrared data communications. 

Common forms of computer-readable media include, for example, a fioppy disk, a flexible 

20 disk, hard disk, magnetic tape, or any other magnetic storage medium, a CD-ROM, any other 
optical medium, punchcards, papertape, any other physical medium with patterns of holes, a 
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RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier 
wave as described hereinafter, or any other medium from which a computer can read and use. 

Various forms of computer readable media may be involved in carrying one or more 
sequences of instructions to processor 1 722 for execution. For example, the instructions may 
5 initially be carried on a magnetic disk or a remote computer. The remote computer can load the 
instructions into its dynamic memory and send the instructions over a telephone line using a 
modem. A modem local to computer system 1712 can receive the data on the telephone line and 
use an infrared transmitter to convert the data to an infrared signal. An infrared detector coupled 
to appropriate circuitry can receive the data carried in the infrared signal and place the data on bus 

1 0 1 726. Bus 1 726 carries the data to main memory 1716, from which processor 1 722 retrieves and 
executes the instructions. The instructions received by main memory 1716 may optionally be 
stored on storage device 1 720 either before or after execution by processor 1 722. 

Computer system 1712 also includes a communication interface 1724 coupled to bus 
1726. Communication interface 1724 provides two way communications to other systems. For 

1 5 example, communication interface 1 724 may be an integrated ser\qccs digital netw^ork (ISDN) 
card or a modem to provide a data communication connection to a corresponding type of 
telephone line. Communication may also be, for example, a local area network (LAN) card to 
provide communication to a LAN. Communication interface 1724 may also be a wireless card for 
implementing wireless communication between computer system 1712 and wireless systems. In 

20 any such implementation, communication interface 1 724 sends and receives electrical, 
electromagnetic or optical signals that carr\^ data streams representing various types of 
information. 
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The link between communication interface 1724 and external devices and systems 
typically provides data communication through one or more networks or other devices. For 
example, the link may provide a connection to a local network (not shown) to a host computer or 
to data equipment operated by an Internet Service Provider (ISP). An ISP provides data 
5 communication services through the world wide packet data communications nctw^ork now 
commonly referred to as the "Internet/' Local netw^orks and the Internet both use electrical, 
electromagnetic or optical signals that carry digital data streams. The signals through the various 
networks and the signals between the networks and communication interface 1724, which caiT>^ 
the digital data to and from computer system 1712, are exemplary forms of carrier waves 

1 0 transporting the information. 

Computer system 1712 can send messages and receive data, including program code, 
through the network(s) via the link between communication interface 1 724 and the external 
systems and devices. In the Internet, for example, a server might transmit a requested code for an 
application program through the Internet, an ISP, a local network, and communication interface 

15 1724. 

Program code received over the network may be executed by processor 1 722 as it is 
received, and/or stored in memory, such as in storage device 1 720, for later execution. In this 
manner, computer system 1712 may obtain application code in the form of a carrier wave. 

Fig. 30 illustrates another embodiment of an image registration system consistent with the 
20 principles of the invention. Camera mouse 1 18 is comprised of camera mouse buttons 110, 112, 
1 14, and a signal cable 120. A micro camera (not shown) in camera mouse 1 1 8 captures the area 
of substrate 122 defined by the target area indicators in response to the observer activating one or 
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more ol camera mouse bullons 110, 112, 114. The image captured by the micro camera can be 
processed locally in mouse 1 1 8 by a processing element (not shown) or sent o\ er signal cable 120 
for processing by a processing element in a device attached to camera mouse 1 1 8. 

Because substrate 122 has embedded data embodied thereon, the embedded data captured 
5 in the image can be decoded. Thus, camera mouse 1 1 8 allows a user to place camera mouse 1 1 8 
over a particular area of substrate 122 and capture an image of the area of substrate 122, which is 
then displayed in display 150. The embedded code in the captured image is decoded by a 
processing element to develop decoded information. The decoded information may represent 
anything about the substrate, such as context, location, and/or orientation. The processing 
10 element analyzes the decoded infomiation and may perform an operation based on the decoded 

information and button controls, or queue the codes and button controls in a syntax queue to aw^ait 
further codes and button controls. 

Based on the decoded information, image infomiation is generated and displayed as an 
image on display 150. Several types of image information may be displayed. For example, the 
1 5 information may comprise the actual image captured by micro camera 1 34, a representation of the 
captured image, the actual image captured in addition to other information related to the captured 
image, or simply information related to the captured image without the captured image. In one 
embodiment, a representation of the substrate is retrieved and combined w ith the captured image 
to form a composite image on display 150 that shows the captured image and the extended area 
20 sun-ounding the captured image. 

Display 150 shows David's Docum icon, which is the captured image, with superimposed 
cross hairs. Display 150 also shows an image 250 of David's Docum, indicating that that icon has 
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been selected by the user. For example, if the user places the mouse over David's Docum, the 
mouse captures the image of David's Docum from substrate 122, and displays the captured image 
on display 1 50. If the user then activates a mouse button to select the icon, a representation of the 
David's Docum icon is displayed in display 150 as Da\'id's Docum icon 250. 
5 Fig. 3 1 is a block diagram showing a cutaway side view of camera mouse 1 1 8. Camera 

mouse 1 18 includes a micro camera 134, an XY motion sensor 136, illuminators 152, and 
processing element 138. Processing element 138 processes signals from XY motion sensor 136 
and images from micro camera 134. In alternative embodiments, XY sensor 136 may be 
removed. 

10 Micro camera 134 receives an image of substrate 122 reflected off semitransparent mirror 

1 16. An observer 130 viewing an image on display 1 50 may capture the image by pressing one or 
more of camera mouse buttons 110, 112, or 1 14. The area of substrate 122 captured by micro 
camera 134 is then processed by processing element 138. Although Fig. 31 illustrates processing 
element 138 located in mouse 1 1 8, all or part of the processing may be performed externally by 

1 5 processing elements connected to camera mouse 1 1 8 by signal cable 1 20. 
Conclusion 

Other embodiments of the invention will be apparent to those skilled in the art from 
consideration of the specification and practice of the disclosed embodiments. The specification 
and examples are exemplary only, and the true scope and spirit of the invention is defined by the 
20 following claims and their equivalents. 
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